Skip to content

perf(db): reduce read/write fanout across hot paths#4704

Merged
waleedlatif1 merged 4 commits into
stagingfrom
waleedlatif1/db-connection-load-optimization
May 21, 2026
Merged

perf(db): reduce read/write fanout across hot paths#4704
waleedlatif1 merged 4 commits into
stagingfrom
waleedlatif1/db-connection-load-optimization

Conversation

@waleedlatif1
Copy link
Copy Markdown
Collaborator

@waleedlatif1 waleedlatif1 commented May 21, 2026

Summary

  • Narrow subscriptionKeys.allusers() invalidations on workflow run / wand / usage-indicator paths; bump useSubscriptionData staleTime 30s → 5min; bump usage-indicator socket debounce 1s → 5s
  • Remove refetchOnMount: 'always' from useDeploymentInfo (defeated 30s staleTime; rely on post-deploy invalidation)
  • Logs page polling 3s → 10s for list + dashboard-stats (detail polling stays 3s when status running/pending)
  • 250ms trailing debounce on tableKeys.activeDispatches SSE invalidations
  • markWebhookSuccess guarded with WHERE failed_count <> 0; webhook PATCH skips no-op writes
  • Collapse 3 SELECT-then-UPDATE/DELETE patterns in MCP orchestration into single statements (also closes a latent workspace-guard gap on UPDATE/DELETE WHERE)
  • Batch N×M credential membership writes with ON CONFLICT DO UPDATE / DO NOTHING; remove unused getPostgresErrorCode helper
  • JSONB column projection sweep across 8 sites (admin workflows/orgs lists, settings GET, template DELETE, template permissions, chat undeploy, copilot deployment status)
  • Narrow .returning() on table row DELETE and copilot getRunSegment
  • Add 4 indexes via migration 0211_breezy_cloak: chat(workflow_id, archived_at), two on webhook, one on workflow_schedule. workflow_execution_logs indexes deliberately deferred to a concurrent-safe deploy (116 GB table)

Type of Change

  • Improvement

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel
Copy link
Copy Markdown

vercel Bot commented May 21, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped May 21, 2026 10:10pm

Request Review

@cursor
Copy link
Copy Markdown

cursor Bot commented May 21, 2026

PR Summary

Medium Risk
Mostly performance-focused query/invalidations changes, but it touches multiple API routes and adds DB indexes/migration; mistakes could cause stale UI data or locking/plan regressions in production.

Overview
Reduces database and cache churn across several hot paths. Many API handlers now project only needed columns (rather than select()), narrow .returning() payloads, and skip unnecessary writes (e.g., no-op webhook PATCH; webhook success update only when failedCount is non-zero).

Cuts client polling/invalidation frequency. Logs list/dashboard live refresh moves from 3s to 10s (detail stays 3s only for running/pending), table SSE dispatch invalidations are debounced, subscription invalidations are narrowed to subscriptionKeys.users() and useSubscriptionData staleTime is increased to 5 minutes; useDeploymentInfo stops forcing refetchOnMount: 'always'.

Tightens and batches server-side writes. MCP lifecycle updates collapse select-then-update into single guarded updates (including server/tool scoping), workspace env credential membership sync is batched via ON CONFLICT upserts/insert-ignore, and migration 0211_breezy_cloak adds indexes on chat, webhook, and workflow_schedule to support the updated access patterns.

Reviewed by Cursor Bugbot for commit 3cf4f7d. Configure here.

Comment thread apps/sim/lib/credentials/environment.ts
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 21, 2026

Greptile Summary

This PR applies a broad performance sweep across hot paths: narrowing query projections (JSONB columns and SELECT *), batching credential membership writes, reducing React Query cache churn, and adding four PostgreSQL indexes for common query patterns. It also closes a latent workspace-guard gap in MCP server UPDATE by folding the check into the single UPDATE…RETURNING statement.

  • DB read reduction: 8+ SELECT * callsites narrowed to projected columns; 3 SELECT-then-UPDATE/DELETE patterns in MCP collapsed to single statements; markWebhookSuccess and webhook PATCH skip writes when no-op; credential membership writes batched with ON CONFLICT DO UPDATE/DO NOTHING
  • Client-side throttling: React Query staleTime bumped from 30 s → 5 min for subscription data; invalidation narrowed from subscriptionKeys.all to subscriptionKeys.users(); logs-page polling 3 s → 10 s; SSE dispatch-invalidate debounced 250 ms
  • Migration 0211_breezy_cloak: adds four indexes on chat, webhook (×2), and workflow_schedule; migration is safe to run against the current table sizes noted in the PR description

Confidence Score: 5/5

Safe to merge — all hot-path optimisations are logically equivalent to the code they replace, workspace guards are preserved or tightened, and the migration targets tables small enough that lock windows are negligible.

The batched credential writes and ON CONFLICT logic match the old per-row loop behaviour. The MCP UPDATE/DELETE collapses include the workspace predicate directly in the WHERE clause and, for the tool update, actually close an old gap where the bare UPDATE had no serverId check. The React Query staleTime bump is paired with explicit invalidation on every mutating operation. No data loss or correctness regression identified.

No files require special attention beyond the minor JSDoc discrepancy in subscription.ts.

Important Files Changed

Filename Overview
apps/sim/lib/credentials/environment.ts Credential membership writes batched with ON CONFLICT DO UPDATE; revoked-member guard preserved via pre-filter; onConflictDoNothing replaces try/catch 23505 pattern for credential inserts
apps/sim/lib/mcp/orchestration/workflow-mcp-lifecycle.ts performUpdateWorkflowMcpServer and performUpdateWorkflowMcpTool collapsed to single UPDATE…RETURNING; workspace guard preserved in WHERE clause; old UPDATE lacked serverId check on tool — now fixed
apps/sim/lib/webhooks/polling/utils.ts markWebhookSuccess now skips the UPDATE when failedCount is already 0 via WHERE failedCount IS NULL OR failedCount != 0
apps/sim/app/api/webhooks/[id]/route.ts PATCH now computes a diff setClause and short-circuits with 200 when no fields actually changed, avoiding a no-op write plus updatedAt bump
packages/db/migrations/0211_breezy_cloak.sql Adds four non-concurrent indexes on small/medium tables (chat 2 MB, workflow_schedule 5 MB, webhook 143 MB); schema.ts kept in sync
apps/sim/hooks/queries/subscription.ts staleTime bumped from 30 s to 5 min; prefetchQuery staleTime updated to match; JSDoc comment not updated to reflect new default
apps/sim/app/workspace/[workspaceId]/tables/[tableId]/hooks/use-table-event-stream.ts 250 ms trailing-edge debounce added for tableKeys.activeDispatches invalidations; timer properly cleaned up on unmount
apps/sim/app/api/v1/admin/types.ts AdminWorkflowSource and AdminOrganizationSource Pick types added to match narrowed DB projections; toAdminWorkflow/toAdminOrganization signatures tightened accordingly
apps/sim/lib/workflows/orchestration/chat-deploy.ts performChatUndeploy SELECT narrowed to the four fields actually consumed downstream (title, workflowId, identifier, authType)

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Workflow Run / Wand / UsageIndicator socket] -->|invalidateQueries| B[subscriptionKeys.users]
    B --> C[useSubscriptionData\nstaleTime=5min]
    C --> D[GET /api/billing user]

    E[SSE dispatch event] -->|scheduleDispatchInvalidate\n250ms debounce| F[tableKeys.activeDispatches]
    F --> G[refetch activeDispatches]

    H[Logs page live mode] -->|refetchInterval 10s| I[logsQuery + dashboardStatsQuery]

    J[PATCH /api/webhooks/:id] --> K{setClause empty?}
    K -->|yes| L[return 200 no-op]
    K -->|no| M[UPDATE webhook]

    N[markWebhookSuccess] --> O{failedCount already 0?}
    O -->|yes, skip| P[no UPDATE]
    O -->|no| Q[UPDATE failedCount=0]

    R[ensureWorkspaceCredentialMemberships] --> S[SELECT existing revoked]
    S --> T[filter out revoked users]
    T --> U[batch INSERT ON CONFLICT DO UPDATE\npreserve joinedAt via COALESCE]

    V[syncWorkspaceEnvCredentials /\ncreateWorkspaceEnvCredentials] --> W[batch INSERT credentials\nON CONFLICT DO NOTHING]
    W --> X[batch INSERT memberships\nON CONFLICT DO NOTHING]
Loading

Reviews (3): Last reviewed commit: "fix(webhooks): include NULL failedCount ..." | Re-trigger Greptile

Comment thread packages/db/migrations/0211_breezy_cloak.sql Outdated
@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

Thanks for the review. Addressed in 4baad33:

  • MCP delete error ambiguity (workflow-mcp-lifecycle.ts:518-529) — restored the server SELECT before the DELETE so a workspace mismatch returns 'Server not found' and a genuinely missing tool returns 'Tool not found'. Matches the pattern used in performUpdateWorkflowMcpTool. The DELETE itself is unchanged (still scoped by id + serverId); the SELECT is a cheap PK lookup so the extra round-trip cost is negligible.
  • chatDeploy projection (manage.ts:45-50) — narrowed to the explicit 9-column projection (id, identifier, title, description, authType, allowedEmails, outputConfigs, password, customizations) matching what's consumed downstream. Aligns with the JSONB sweep theme.

@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread apps/sim/lib/webhooks/polling/utils.ts Outdated
@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@cursor review

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 3cf4f7d. Configure here.

@waleedlatif1 waleedlatif1 merged commit 3f7698c into staging May 21, 2026
14 checks passed
@waleedlatif1 waleedlatif1 deleted the waleedlatif1/db-connection-load-optimization branch May 21, 2026 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant